Web API Processing

The pattern of calling a Web API separates across two stages:

  1. Construct a set of web request objects
  2. Execute for each one of those objects

For instance, a simple call against a public web resource might look like:

Step 1: Create an array containing one request:

This could be a mapping process applied to an incoming payload, or it could start a process anew, not relying on any input.
The simplest thing is to start it anew, with a jsDataStreamCreator. This processor always runs, regardless of whether it is passed any data on the pipeline. It can return any kind of data, making it the most flexible of the processors.
return [ new HttpRequest( 'https', 'www.cat-fact.herokuapp.com', Some(443), newList(['facts']), 'GET') ]

Step 2: Execute all the requests in that array:

There is no script in this step. An HTTP request executor is simply a processor of type HttpStreamProviderConfiguration.
This processor type includes configuration fields such as:
  • retry limit
  • retry delay in miliseconds
  • which certificate authorities to trust
  • how many requests to run in parallel
  • whether to validate certificates for HTTPS
  • a throttling limit for total requests per minute

In this example, we've set the host within the request construction. This is the most general pattern. For a dedicated processor, we'll provide the host within the executor configuration instead of the request.